Formalizing knowledge used in spectrogram reading: acoustic and perceptual evidence from stops

Since the invention of the sound spectrograph in 1946 by Koenig, Dunn and Lacey, spectrograms have been widely used for speech research. Over the last decade there has been revived interest in the application of spectrogram reading toward continuous speech recognition. Spectrogram reading involves interpreting the acoustic patterns in the image to determine the spoken utterance. One must selectively attend to many different acoustic cues, interpret their significance in light of other evidence, and make inferences based on information from multiple sources. While early attempts at spectrogram reading met with limited success (Klatt and Stevens, 1973; Lindblom and Svenssen, 1973; Svenssen, 1974), Zue, in a series of experiments intended to illustrate the richness of phonetic information in the speech signal (Cole et al., 1980; Cole and Zue, 1980), demonstrated that high performance phonetic labeling of a spectrogram could be obtained. In this thesis a formal evaluation of spectrogram reading was conducted in order to obtain a better understanding of the process and to evaluate the ability of spectrogram readers. The research consisted of three main parts: an evaluation of spectrogram readers on a constrained task, a comparison to listeners on the same task, and a formalization of spectrogram-reading knowledge in a rule-based system. The performance of 5 spectrogram readers was assessed using speech from 299 talkers. The readers identified stop consonants which were extracted from continuous speech and presented in the immediate phonemic context. The task was designed so that lexical and other higher sources of knowledge could not be used. The averaged identification rate of the ranged across contexts, from 73-82% top choice, and 77-93% for the top two choices. The performance of spectrogram readers was, on the average, 10% below that of human listeners on the same task. Listeners had an overall identification rate that ranged from 85 to 97%. The performance of readers is comparable to other spectrogram reading experiments reported in the literature, however the other studies have typically evaluated a single subject on speech spoken by a small number of talkers. Although researchers have suggested that the process can be described in terms of rules (Zue, 1981), few compilations of rules or strategies exist (Rothenberg, 1963; Fant, 1968, Svenssen, 1974). In order to formalize the information used in spectrogram reading, a system for identifying stop consonants was developed. A knowledge-based system was chosen because the expression and use of the knowledge is explicit. The emphasis was on capturing the acoustic descriptions and modeling the reasoning thought to be used